Check for failures in DT CI #10544

0xOmarA · 2025-12-04T14:59:08Z

Description

This is a small PR that makes the CI of the differential testing framework fail if any failure is encountered. We only do this with the revive-dev-node-revm-solc target since this is the only target at the moment that has no failures.

0xOmarA · 2025-12-04T14:59:49Z

/cmd prdoc --audience runtime_dev

…time_dev'

0xRVE · 2025-12-04T15:22:33Z

.github/scripts/differential-tests-error-out-on-failures.py

Might be better to us jq instead of a python script

I know that the Python script is a bit longer than JQ, but I think that it's more readable than it when it comes to what we're trying to do here :)

Wouldn't that end up in a pretty unreadable CLI?

I didn't try it but I think it should be a simple oneliner.

Better have the test-suite return status e.g. number of tests failed -> 0 means success. IF that is easy to make

I don't really care
but this do seems to be a bit of an overkill if we can just do

report_file="$1" if jq -e ' any( .execution_information[] | .case_reports[] | .mode_execution_reports[] | .status.status; . == "Failed" ) ' $report_file; then exit 1 else exit 0 fi

athei

Why do we have to have a 87 line parsing script to determine if the test suite succeeded? Can't we just return proper status codes when calling the test suite?

athei · 2025-12-04T15:26:36Z

.github/scripts/differential-tests-error-out-on-failures.py

Wouldn't that end up in a pretty unreadable CLI?

pgherveou · 2025-12-04T15:51:04Z

.github/scripts/differential-tests-error-out-on-failures.py

I don't really care
but this do seems to be a bit of an overkill if we can just do

report_file="$1" if jq -e ' any( .execution_information[] | .case_reports[] | .mode_execution_reports[] | .status.status; . == "Failed" ) ' $report_file; then exit 1 else exit 0 fi

0xOmarA · 2025-12-05T03:03:03Z

@athei

Why do we have to have a 87 line parsing script to determine if the test suite succeeded? Can't we just return proper status codes when calling the test suite?

In the future we could. But for the time being, we expect some of the tests in the PolkaVM test suite to fail and we expect all of the tests in the REVM test suite to succeed. So, I think that making retester return a status code based on failures is very doable but I think it makes more sense once all of the tests for REVM and PolkaVM pass and we no longer need to have this conditional logic.

Also, a very big portion of this script is just models that I copied over from another script that I wrote. We technically don't need this whole code, all that we need is the following:

with open(sys.argv[1], "r") as file:
    report: Report = json.load(file)

for _, mode_to_case_mapping in report["execution_information"].items():
    for _, case_idx_to_report_mapping in mode_to_case_mapping[
        "case_reports"
    ].items():
        for _, execution_report in case_idx_to_report_mapping[
            "mode_execution_reports"
        ].items():
            if execution_report["status"]["status"] == "Failed":
                exit(1)

exit(0)

github-actions · 2025-12-05T07:59:59Z

Differential Tests Results (PolkaVM)

Specified Tests

simple
complex
translated_semantic_tests

Counts

Total Number of Test Cases: 26447
Total Number of Successes: 5797
Total Number of Failures: 58
Total Number of Ignores: 20592

Failures

The test specifiers seen in this section have the format 'path::case_idx::compilation_mode' and they're compatible with the revive differential tests framework and can be specified to it directly in the same way that they're provided through the --test argument of the framework.

The failures are provided in an expandable section to ensure that the PR does not get polluted with information. Please click on the section below for more information

Detailed Differential Tests Failure Information

Test Specifier	Failure Reason	Note
`complex/create/create_in_library/test.json::0::Y+`	`Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected true but got false. Revert reason: Some("")`
`complex/create/create_in_library/test.json::0::Y-`	`Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected true but got false. Revert reason: Some("")`
`complex/library_call_tuple/test.json::0::Y-`	`Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected true but got false. Revert reason: Some("")`
`complex/library_call_tuple/test.json::0::Y+`	`Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected true but got false. Revert reason: Some("")`
`complex/solidity_by_example/applications/iterable_mapping/test.json::0::Y- >=0.8.1`	`Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected true but got false. Revert reason: Some("")`
`complex/solidity_by_example/applications/iterable_mapping/test.json::0::Y+ >=0.8.1`	`Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected true but got false. Revert reason: Some("")`
`simple/function/many_arguments_2.sol::0::Y-`	Failed to create the drivers for the various platforms: Failed to create driver for revive-dev-node-polkavm-resolc: Failed to initialize the execution state of the platform: Failed to produce the pre-linking compiled contracts: Compilation callback failed (cache miss path): Encountered an error in the compilation: CompilerError: Stack too deep. Try compiling with `--via-ir` (cli) or the equivalent `viaIR: true` (standard JSON) while enabling the optimizer. Otherwise, try removing local variables. --> /__w/polkadot-sdk/polkadot-sdk/revive-differential-tests/resolc-compiler-tests/fixtures/solidity/simple/function/many_arguments_2.sol:52:40: 52 result = result && main(i, p1, p2, p3, p4, p5, p6, p7, p8, p9, p10, p11, p12, p13, p14, p15, p16, p17, p18, p19, p20) == uint256(i) * i; ^^
`simple/function/many_arguments_2.sol::0::Y+`	Failed to create the drivers for the various platforms: Failed to create driver for revive-dev-node-polkavm-resolc: Failed to initialize the execution state of the platform: Failed to produce the pre-linking compiled contracts: Compilation callback failed (cache miss path): Encountered an error in the compilation: CompilerError: Stack too deep. Try compiling with `--via-ir` (cli) or the equivalent `viaIR: true` (standard JSON) while enabling the optimizer. Otherwise, try removing local variables. --> /__w/polkadot-sdk/polkadot-sdk/revive-differential-tests/resolc-compiler-tests/fixtures/solidity/simple/function/many_arguments_2.sol:52:40: 52 result = result && main(i, p1, p2, p3, p4, p5, p6, p7, p8, p9, p10, p11, p12, p13, p14, p15, p16, p17, p18, p19, p20) == uint256(i) * i; ^^
`simple/function/many_arguments_2_complex.sol::0::Y+`	Failed to create the drivers for the various platforms: Failed to create driver for revive-dev-node-polkavm-resolc: Failed to initialize the execution state of the platform: Failed to produce the pre-linking compiled contracts: Compilation callback failed (cache miss path): Encountered an error in the compilation: CompilerError: Stack too deep. Try compiling with `--via-ir` (cli) or the equivalent `viaIR: true` (standard JSON) while enabling the optimizer. Otherwise, try removing local variables. --> /__w/polkadot-sdk/polkadot-sdk/revive-differential-tests/resolc-compiler-tests/fixtures/solidity/simple/function/many_arguments_2_complex.sol:58:21: 58 p1, ^^
`simple/internal_function_pointers/sum_oddness.sol::0::Y+`	`Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected true but got false. Revert reason: Some("ContractTrapped")`	This test case succeeded with other compilation modes: {'Y-'}
`simple/internal_function_pointers/sum_oddness.sol::1::Y+`	`Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected true but got false. Revert reason: Some("ContractTrapped")`	This test case succeeded with other compilation modes: {'Y-'}
`simple/pointer/large_offset.sol::0::Y+`	`Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected true but got false. Revert reason: Some("ContractTrapped")`
`simple/pointer/large_offset.sol::0::Y-`	`Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected true but got false. Revert reason: Some("ContractTrapped")`
`simple/recursion/recursion_keccak.sol::0::Y+`	`Failed to create the drivers for the various platforms: Failed to create driver for revive-dev-node-polkavm-resolc: Failed to initialize the execution state of the platform: Failed to produce the pre-linking compiled contracts: Compilation callback failed (cache miss path): Failed to parse resolc standard JSON output: failed to parse resolc JSON output: recursion limit exceeded at line 1 column 184228 stderr:`
`simple/recursion/recursion_keccak.sol::0::Y-`	`Failed to create the drivers for the various platforms: Failed to create driver for revive-dev-node-polkavm-resolc: Failed to initialize the execution state of the platform: Failed to produce the pre-linking compiled contracts: Compilation callback failed (cache miss path): Failed to parse resolc standard JSON output: failed to parse resolc JSON output: recursion limit exceeded at line 1 column 267625 stderr:`
`simple/recursion_keccak.sol::0::Y+`	`Failed to create the drivers for the various platforms: Failed to create driver for revive-dev-node-polkavm-resolc: Failed to initialize the execution state of the platform: Failed to produce the pre-linking compiled contracts: Compilation callback failed (cache miss path): Failed to parse resolc standard JSON output: failed to parse resolc JSON output: recursion limit exceeded at line 1 column 184158 stderr:`
`simple/recursion_keccak.sol::0::Y-`	`Failed to create the drivers for the various platforms: Failed to create driver for revive-dev-node-polkavm-resolc: Failed to initialize the execution state of the platform: Failed to produce the pre-linking compiled contracts: Compilation callback failed (cache miss path): Failed to parse resolc standard JSON output: failed to parse resolc JSON output: recursion limit exceeded at line 1 column 267555 stderr:`
`simple/try_catch/unbalanced_gas_limit.sol::0::Y+`	Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Output assertion failed - Expected Compound([CalldataItem("0"), CalldataItem("1"), CalldataItem("0"), CalldataItem("0")]) but got 0x0000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001
`simple/try_catch/unbalanced_gas_limit.sol::0::Y-`	Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Output assertion failed - Expected Compound([CalldataItem("0"), CalldataItem("1"), CalldataItem("0"), CalldataItem("0")]) but got 0x0000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001
`simple/yul_instructions/returndatacopy.sol::41::Y-`	`Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected true but got false. Revert reason: Some("OutOfGas")`
`simple/yul_instructions/returndatacopy.sol::41::Y+`	`Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected true but got false. Revert reason: Some("OutOfGas")`
`simple/yul_instructions/revert.sol::59::Y+`	`Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Output assertion failed - Expected Compound([CalldataItem("0xdeadbeef")]) but got 0x0000000000000000000000000000000000000000000000000000000000000000`
`simple/yul_instructions/revert.sol::59::Y-`	`Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Output assertion failed - Expected Compound([CalldataItem("0xdeadbeef")]) but got 0x0000000000000000000000000000000000000000000000000000000000000000`
`simple/yul_instructions/revert.sol::60::Y-`	`Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Output assertion failed - Expected Compound([CalldataItem("0xdeadbeef")]) but got 0x0000000000000000000000000000000000000000000000000000000000000000`
`simple/yul_instructions/revert.sol::60::Y+`	`Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Output assertion failed - Expected Compound([CalldataItem("0xdeadbeef")]) but got 0x0000000000000000000000000000000000000000000000000000000000000000`
`simple/yul_instructions/revert.sol::61::Y+`	`Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Output assertion failed - Expected Compound([CalldataItem("0xdeadbeef")]) but got 0x0000000000000000000000000000000000000000000000000000000000000000`
`simple/yul_instructions/revert.sol::61::Y-`	`Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Output assertion failed - Expected Compound([CalldataItem("0xdeadbeef")]) but got 0x0000000000000000000000000000000000000000000000000000000000000000`
`simple/yul_instructions/revert.sol::75::Y-`	`Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Output assertion failed - Expected Compound([CalldataItem("0xdeadbeef")]) but got 0x0000000000000000000000000000000000000000000000000000000000000000`
`simple/yul_instructions/revert.sol::75::Y+`	`Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Output assertion failed - Expected Compound([CalldataItem("0xdeadbeef")]) but got 0x0000000000000000000000000000000000000000000000000000000000000000`
`simple/yul_instructions/revert.sol::197::Y-`	`Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Output assertion failed - Expected Compound([CalldataItem("0xdeadbeef")]) but got 0x0000000000000000000000000000000000000000000000000000000000000000`
`simple/yul_instructions/revert.sol::197::Y+`	`Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Output assertion failed - Expected Compound([CalldataItem("0xdeadbeef")]) but got 0x0000000000000000000000000000000000000000000000000000000000000000`
`simple/yul_instructions/revert.sol::200::Y+`	`Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected true but got false. Revert reason: Some("OutOfGas")`
`simple/yul_instructions/revert.sol::200::Y-`	`Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected true but got false. Revert reason: Some("OutOfGas")`
`simple/yul_instructions/revert.sol::202::Y-`	`Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Output assertion failed - Expected Compound([CalldataItem("0xdeadbeef")]) but got 0x0000000000000000000000000000000000000000000000000000000000000000`
`simple/yul_instructions/revert.sol::202::Y+`	`Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Output assertion failed - Expected Compound([CalldataItem("0xdeadbeef")]) but got 0x0000000000000000000000000000000000000000000000000000000000000000`
`simple/yul_instructions/revert.sol::205::Y+`	`Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected true but got false. Revert reason: Some("OutOfGas")`
`simple/yul_instructions/revert.sol::205::Y-`	`Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected true but got false. Revert reason: Some("OutOfGas")`
`simple/yul_instructions/revert.sol::207::Y-`	`Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Output assertion failed - Expected Compound([CalldataItem("0xdeadbeef")]) but got 0x0000000000000000000000000000000000000000000000000000000000000000`
`simple/yul_instructions/revert.sol::207::Y+`	`Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Output assertion failed - Expected Compound([CalldataItem("0xdeadbeef")]) but got 0x0000000000000000000000000000000000000000000000000000000000000000`
`simple/yul_instructions/revert.sol::210::Y+`	`Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected true but got false. Revert reason: Some("OutOfGas")`
`simple/yul_instructions/revert.sol::210::Y-`	`Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected true but got false. Revert reason: Some("OutOfGas")`
`translated_semantic_tests/array/copying/array_nested_calldata_to_storage/test.json::0::Y-`	Failed to create the drivers for the various platforms: Failed to create driver for revive-dev-node-polkavm-resolc: Failed to initialize the execution state of the platform: Failed to produce the pre-linking compiled contracts: Compilation callback failed (cache miss path): Encountered an error in the compilation: UnimplementedFeatureError: Copying nested calldata dynamic arrays to storage is not implemented in the old code generator. --> /__w/polkadot-sdk/polkadot-sdk/revive-differential-tests/resolc-compiler-tests/fixtures/solidity/translated_semantic_tests/array/copying/array_nested_calldata_to_storage/array_nested_calldata_to_storage.sol:2:1: 2 contract c { ^ (Relevant source part starts here and spans across multiple lines).
`translated_semantic_tests/array/copying/array_of_struct_calldata_to_storage/test.json::0::Y-`	Failed to create the drivers for the various platforms: Failed to create driver for revive-dev-node-polkavm-resolc: Failed to initialize the execution state of the platform: Failed to produce the pre-linking compiled contracts: Compilation callback failed (cache miss path): Encountered an error in the compilation: UnimplementedFeatureError: Copying of type struct C.S calldata[] calldata to storage is not supported in legacy (only supported by the IR pipeline). Hint: try compiling with `--via-ir` (CLI) or the equivalent `viaIR: true` (Standard JSON) --> /__w/polkadot-sdk/polkadot-sdk/revive-differential-tests/resolc-compiler-tests/fixtures/solidity/translated_semantic_tests/array/copying/array_of_struct_calldata_to_storage/array_of_struct_calldata_to_storage.sol:2:1: 2 contract C { ^ (Relevant source part starts here and spans across multiple lines).
`translated_semantic_tests/array/copying/array_of_struct_memory_to_storage/test.json::0::Y-`	Failed to create the drivers for the various platforms: Failed to create driver for revive-dev-node-polkavm-resolc: Failed to initialize the execution state of the platform: Failed to produce the pre-linking compiled contracts: Compilation callback failed (cache miss path): Encountered an error in the compilation: UnimplementedFeatureError: Copying of type struct C.S memory[] memory to storage is not supported in legacy (only supported by the IR pipeline). Hint: try compiling with `--via-ir` (CLI) or the equivalent `viaIR: true` (Standard JSON) --> /__w/polkadot-sdk/polkadot-sdk/revive-differential-tests/resolc-compiler-tests/fixtures/solidity/translated_semantic_tests/array/copying/array_of_struct_memory_to_storage/array_of_struct_memory_to_storage.sol:1:1: 1 contract C { ^ (Relevant source part starts here and spans across multiple lines).
`translated_semantic_tests/array/copying/array_of_structs_containing_arrays_calldata_to_storage/test.json::0::Y-`	Failed to create the drivers for the various platforms: Failed to create driver for revive-dev-node-polkavm-resolc: Failed to initialize the execution state of the platform: Failed to produce the pre-linking compiled contracts: Compilation callback failed (cache miss path): Encountered an error in the compilation: UnimplementedFeatureError: Copying of type struct C.S calldata[] calldata to storage is not supported in legacy (only supported by the IR pipeline). Hint: try compiling with `--via-ir` (CLI) or the equivalent `viaIR: true` (Standard JSON) --> /__w/polkadot-sdk/polkadot-sdk/revive-differential-tests/resolc-compiler-tests/fixtures/solidity/translated_semantic_tests/array/copying/array_of_structs_containing_arrays_calldata_to_storage/array_of_structs_containing_arrays_calldata_to_storage.sol:2:1: 2 contract C { ^ (Relevant source part starts here and spans across multiple lines).
`translated_semantic_tests/array/copying/array_of_structs_containing_arrays_memory_to_storage/test.json::0::Y-`	Failed to create the drivers for the various platforms: Failed to create driver for revive-dev-node-polkavm-resolc: Failed to initialize the execution state of the platform: Failed to produce the pre-linking compiled contracts: Compilation callback failed (cache miss path): Encountered an error in the compilation: UnimplementedFeatureError: Copying of type struct C.S memory[] memory to storage is not supported in legacy (only supported by the IR pipeline). Hint: try compiling with `--via-ir` (CLI) or the equivalent `viaIR: true` (Standard JSON) --> /__w/polkadot-sdk/polkadot-sdk/revive-differential-tests/resolc-compiler-tests/fixtures/solidity/translated_semantic_tests/array/copying/array_of_structs_containing_arrays_memory_to_storage/array_of_structs_containing_arrays_memory_to_storage.sol:2:1: 2 contract C { ^ (Relevant source part starts here and spans across multiple lines).
`translated_semantic_tests/array/copying/calldata_array_to_mapping/test.json::0::Y-`	Failed to create the drivers for the various platforms: Failed to create driver for revive-dev-node-polkavm-resolc: Failed to initialize the execution state of the platform: Failed to produce the pre-linking compiled contracts: Compilation callback failed (cache miss path): Encountered an error in the compilation: UnimplementedFeatureError: Copying nested calldata dynamic arrays to storage is not implemented in the old code generator. --> /__w/polkadot-sdk/polkadot-sdk/revive-differential-tests/resolc-compiler-tests/fixtures/solidity/translated_semantic_tests/array/copying/calldata_array_to_mapping/calldata_array_to_mapping.sol:2:1: 2 contract C { ^ (Relevant source part starts here and spans across multiple lines).
`translated_semantic_tests/array/copying/elements_of_nested_array_of_structs_calldata_to_storage/test.json::0::Y-`	Failed to create the drivers for the various platforms: Failed to create driver for revive-dev-node-polkavm-resolc: Failed to initialize the execution state of the platform: Failed to produce the pre-linking compiled contracts: Compilation callback failed (cache miss path): Encountered an error in the compilation: UnimplementedFeatureError: Copying nested calldata dynamic arrays to storage is not implemented in the old code generator. --> /__w/polkadot-sdk/polkadot-sdk/revive-differential-tests/resolc-compiler-tests/fixtures/solidity/translated_semantic_tests/array/copying/elements_of_nested_array_of_structs_calldata_to_storage/elements_of_nested_array_of_structs_calldata_to_storage.sol:2:1: 2 contract C { ^ (Relevant source part starts here and spans across multiple lines).
`translated_semantic_tests/array/copying/elements_of_nested_array_of_structs_memory_to_storage/test.json::0::Y-`	Failed to create the drivers for the various platforms: Failed to create driver for revive-dev-node-polkavm-resolc: Failed to initialize the execution state of the platform: Failed to produce the pre-linking compiled contracts: Compilation callback failed (cache miss path): Encountered an error in the compilation: UnimplementedFeatureError: Copying of type struct C.S memory[1] memory to storage is not supported in legacy (only supported by the IR pipeline). Hint: try compiling with `--via-ir` (CLI) or the equivalent `viaIR: true` (Standard JSON) --> /__w/polkadot-sdk/polkadot-sdk/revive-differential-tests/resolc-compiler-tests/fixtures/solidity/translated_semantic_tests/array/copying/elements_of_nested_array_of_structs_memory_to_storage/elements_of_nested_array_of_structs_memory_to_storage.sol:2:1: 2 contract C { ^ (Relevant source part starts here and spans across multiple lines).
`translated_semantic_tests/array/copying/nested_array_of_structs_calldata_to_storage/test.json::0::Y-`	Failed to create the drivers for the various platforms: Failed to create driver for revive-dev-node-polkavm-resolc: Failed to initialize the execution state of the platform: Failed to produce the pre-linking compiled contracts: Compilation callback failed (cache miss path): Encountered an error in the compilation: UnimplementedFeatureError: Copying nested calldata dynamic arrays to storage is not implemented in the old code generator. --> /__w/polkadot-sdk/polkadot-sdk/revive-differential-tests/resolc-compiler-tests/fixtures/solidity/translated_semantic_tests/array/copying/nested_array_of_structs_calldata_to_storage/nested_array_of_structs_calldata_to_storage.sol:2:1: 2 contract C { ^ (Relevant source part starts here and spans across multiple lines).
`translated_semantic_tests/array/copying/nested_array_of_structs_memory_to_storage/test.json::0::Y-`	Failed to create the drivers for the various platforms: Failed to create driver for revive-dev-node-polkavm-resolc: Failed to initialize the execution state of the platform: Failed to produce the pre-linking compiled contracts: Compilation callback failed (cache miss path): Encountered an error in the compilation: UnimplementedFeatureError: Copying of type struct C.S memory[] memory to storage is not supported in legacy (only supported by the IR pipeline). Hint: try compiling with `--via-ir` (CLI) or the equivalent `viaIR: true` (Standard JSON) --> /__w/polkadot-sdk/polkadot-sdk/revive-differential-tests/resolc-compiler-tests/fixtures/solidity/translated_semantic_tests/array/copying/nested_array_of_structs_memory_to_storage/nested_array_of_structs_memory_to_storage.sol:2:1: 2 contract C { ^ (Relevant source part starts here and spans across multiple lines).
`translated_semantic_tests/array/copying/nested_array_of_structs_storage_to_storage/test.json::0::Y-`	Failed to create the drivers for the various platforms: Failed to create driver for revive-dev-node-polkavm-resolc: Failed to initialize the execution state of the platform: Failed to produce the pre-linking compiled contracts: Compilation callback failed (cache miss path): Encountered an error in the compilation: UnimplementedFeatureError: Copying of type struct C.S memory[1] memory to storage is not supported in legacy (only supported by the IR pipeline). Hint: try compiling with `--via-ir` (CLI) or the equivalent `viaIR: true` (Standard JSON) --> /__w/polkadot-sdk/polkadot-sdk/revive-differential-tests/resolc-compiler-tests/fixtures/solidity/translated_semantic_tests/array/copying/nested_array_of_structs_storage_to_storage/nested_array_of_structs_storage_to_storage.sol:2:1: 2 contract C { ^ (Relevant source part starts here and spans across multiple lines).
`translated_semantic_tests/array/copying/nested_array_of_structs_with_nested_array_from_storage_to_memory/test.json::0::Y-`	Failed to create the drivers for the various platforms: Failed to create driver for revive-dev-node-polkavm-resolc: Failed to initialize the execution state of the platform: Failed to produce the pre-linking compiled contracts: Compilation callback failed (cache miss path): Encountered an error in the compilation: UnimplementedFeatureError: Copying of type struct C.S memory[] memory to storage is not supported in legacy (only supported by the IR pipeline). Hint: try compiling with `--via-ir` (CLI) or the equivalent `viaIR: true` (Standard JSON) --> /__w/polkadot-sdk/polkadot-sdk/revive-differential-tests/resolc-compiler-tests/fixtures/solidity/translated_semantic_tests/array/copying/nested_array_of_structs_with_nested_array_from_storage_to_memory/nested_array_of_structs_with_nested_array_from_storage_to_memory.sol:2:1: 2 contract C { ^ (Relevant source part starts here and spans across multiple lines).
`translated_semantic_tests/array/copying/nested_dynamic_array_element_calldata_to_storage/test.json::0::Y-`	Failed to create the drivers for the various platforms: Failed to create driver for revive-dev-node-polkavm-resolc: Failed to initialize the execution state of the platform: Failed to produce the pre-linking compiled contracts: Compilation callback failed (cache miss path): Encountered an error in the compilation: UnimplementedFeatureError: Copying nested calldata dynamic arrays to storage is not implemented in the old code generator. --> /__w/polkadot-sdk/polkadot-sdk/revive-differential-tests/resolc-compiler-tests/fixtures/solidity/translated_semantic_tests/array/copying/nested_dynamic_array_element_calldata_to_storage/nested_dynamic_array_element_calldata_to_storage.sol:2:1: 2 contract C { ^ (Relevant source part starts here and spans across multiple lines).
`translated_semantic_tests/array/nested_calldata_storage/test.json::0::Y-`	Failed to create the drivers for the various platforms: Failed to create driver for revive-dev-node-polkavm-resolc: Failed to initialize the execution state of the platform: Failed to produce the pre-linking compiled contracts: Compilation callback failed (cache miss path): Encountered an error in the compilation: UnimplementedFeatureError: Copying nested calldata dynamic arrays to storage is not implemented in the old code generator. --> /__w/polkadot-sdk/polkadot-sdk/revive-differential-tests/resolc-compiler-tests/fixtures/solidity/translated_semantic_tests/array/nested_calldata_storage/nested_calldata_storage.sol:2:1: 2 contract C { ^ (Relevant source part starts here and spans across multiple lines).
`translated_semantic_tests/array/nested_calldata_storage2/test.json::0::Y-`	Failed to create the drivers for the various platforms: Failed to create driver for revive-dev-node-polkavm-resolc: Failed to initialize the execution state of the platform: Failed to produce the pre-linking compiled contracts: Compilation callback failed (cache miss path): Encountered an error in the compilation: UnimplementedFeatureError: Copying nested calldata dynamic arrays to storage is not implemented in the old code generator. --> /__w/polkadot-sdk/polkadot-sdk/revive-differential-tests/resolc-compiler-tests/fixtures/solidity/translated_semantic_tests/array/nested_calldata_storage2/nested_calldata_storage2.sol:2:1: 2 contract C { ^ (Relevant source part starts here and spans across multiple lines).
`translated_semantic_tests/saltedCreate/prediction_example/test.json::0::Y-`	`Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected true but got false. Revert reason: Some("revert: Address mismatch.")`
`translated_semantic_tests/shanghai/evmone_support/test.json::0::Y-`	Failed to create the drivers for the various platforms: Failed to create driver for revive-dev-node-polkavm-resolc: Failed to initialize the execution state of the platform: Failed to produce the pre-linking compiled contracts: Compilation callback failed (cache miss path): Encountered an error in the compilation: Error: LLVM IR generator: 306:17 The `EXTCODECOPY` instruction is not supported --> /__w/polkadot-sdk/polkadot-sdk/revive-differential-tests/resolc-compiler-tests/fixtures/solidity/translated_semantic_tests/shanghai/evmone_support/evmone_support.sol

github-actions · 2025-12-05T08:01:28Z

Differential Tests Results (REVM)

Specified Tests

simple
complex
translated_semantic_tests

Counts

Total Number of Test Cases: 26447
Total Number of Successes: 19161
Total Number of Failures: 0
Total Number of Ignores: 7286

Failures

The test specifiers seen in this section have the format 'path::case_idx::compilation_mode' and they're compatible with the revive differential tests framework and can be specified to it directly in the same way that they're provided through the --test argument of the framework.

The failures are provided in an expandable section to ensure that the PR does not get polluted with information. Please click on the section below for more information

Detailed Differential Tests Failure Information

Test Specifier	Failure Reason	Note

paritytech-workflow-stopper · 2025-12-05T08:38:33Z

All GitHub workflows were cancelled due to failure one of the required jobs.
Failed workflow url: https://github.com/paritytech/polkadot-sdk/actions/runs/19956350549
Failed job name: check-runtime-migration

athei · 2025-12-08T14:31:50Z

So, I think that making retester return a status code based on failures is very doable but I think it makes more sense once all of the tests for REVM and PolkaVM pass and we no longer need to have this conditional logic.

I don't get how this is related. The CI script would allow non zero status code for PVM but not for EVM. That is exactly what we are doing here just with extra steps (the parsing). We could also return the number of failing tests as status code. Determining whether the tests were OK or FAIL should easy.

Check for failures in DT Ci

9cfebb3

0xOmarA requested review from a team as code owners December 4, 2025 14:59

0xOmarA added the T7-smart_contracts This PR/Issue is related to smart contracts. label Dec 4, 2025

github-actions bot and others added 3 commits December 4, 2025 15:02

Update from github-actions[bot] running command 'prdoc --audience run…

c096a75

…time_dev'

Remove un-needed comment

2c48ad2

Update doc comment

9a9e1ac

0xRVE reviewed Dec 4, 2025

View reviewed changes

athei reviewed Dec 4, 2025

View reviewed changes

.github/scripts/differential-tests-error-out-on-failures.py

Copy link

Member

athei Dec 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't that end up in a pretty unreadable CLI?

pgherveou approved these changes Dec 4, 2025

View reviewed changes

Update the commit hash of the DT framework

f864865

Check for failures in DT CI #10544

Are you sure you want to change the base?

Check for failures in DT CI #10544

Conversation

0xOmarA commented Dec 4, 2025

Description

Uh oh!

0xOmarA commented Dec 4, 2025

Uh oh!

0xRVE Dec 4, 2025

Choose a reason for hiding this comment

Uh oh!

0xOmarA Dec 4, 2025

Choose a reason for hiding this comment

Uh oh!

athei Dec 4, 2025

Choose a reason for hiding this comment

Uh oh!

0xRVE Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pgherveou Dec 4, 2025

Choose a reason for hiding this comment

Uh oh!

athei left a comment

Choose a reason for hiding this comment

Uh oh!

athei Dec 4, 2025

Choose a reason for hiding this comment

Uh oh!

pgherveou Dec 4, 2025

Choose a reason for hiding this comment

Uh oh!

0xOmarA commented Dec 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Dec 5, 2025

Differential Tests Results (PolkaVM)

Specified Tests

Counts

Failures

Uh oh!

github-actions bot commented Dec 5, 2025

Differential Tests Results (REVM)

Specified Tests

Counts

Failures

Uh oh!

paritytech-workflow-stopper bot commented Dec 5, 2025

Uh oh!

athei commented Dec 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

0xRVE Dec 4, 2025 •

edited

Loading

0xOmarA commented Dec 5, 2025 •

edited

Loading

athei commented Dec 8, 2025 •

edited

Loading